{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Edge Probing Side-by-Side Examples\n",
    "\n",
    "This notebook is designed to load predictions from two runs and mine for interesting win or loss examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys, os, re, json\n",
    "from importlib import reload\n",
    "\n",
    "from src.utils import utils"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tqdm import tqdm\n",
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<module 'analysis' from '/nfs/jsalt/home/iftenney/jiant_test/probing/analysis.py'>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import analysis; reload(analysis)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_raw_preds(task_name, exp_type, split_name=\"val\"):\n",
    "    preds_path = f\"{project_dir}/{exp_type}-{task_name}/run/{task_name}_{split_name}.json\"\n",
    "    return list(utils.load_json_data(preds_path))\n",
    "\n",
    "def load_task_preds(task_name, exp_type, split_name=\"val\"):\n",
    "    run_dir = f\"{project_dir}/{exp_type}-{task_name}/run\"\n",
    "    return analysis.Predictions.from_run(run_dir, task_name, split_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the cell below to configure what to load:\n",
    "\n",
    "- `project_dir` should be the path to a directory of experiments\n",
    "- `exp_types` should be the two experiment types to compare\n",
    "- `task_name` is the task to look at\n",
    "- `split_name` is the split (`val` or `test`) to look at\n",
    "\n",
    "At minimum, you'll want to point `project_dir` to something available on your system.\n",
    "\n",
    "This assumes that the project directory contains experiments named as `{exp_type}-{task_name}`, each containing a single run named `run`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# project_dir = \"/nfs/jsalt/exp/edges-20180926-elmofix\"\n",
    "# exp_types = [\"elmo-chars\", \"elmo-full\"]\n",
    "# project_dir = \"/nfs/jsalt/exp/edges-20190124-bert\"\n",
    "# exp_types = [\"bert-base-uncased-lex\", \"bert-base-uncased-cat\"]\n",
    "project_dir = \"/nfs/jsalt/home/iftenney/exp/bert_mix_20190129\"\n",
    "exp_types = [\"bert-large-uncased-lex\", \"bert-large-uncased-mix\"]\n",
    "\n",
    "# task_name = \"edges-srl-conll2012\"\n",
    "# task_name = \"edges-spr2\"\n",
    "task_name = \"edges-coref-ontonotes-conll\"\n",
    "split_name = \"val\"  # look at development / validation sets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll load both the raw predictions (records loaded from JSON), and also process them (with `load_task_preds`, which uses `analysis.Predictions`) into a long-form DataFrame. We can use the DataFrame to easily score groups of targets, and then retrieve the full records from the raw predictions based on the example index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "r0 = load_raw_preds(task_name, exp_types[0])\n",
    "r1 = load_raw_preds(task_name, exp_types[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "02/09/2019 02:25:52 - INFO - root -   Loading vocabulary from /nfs/jsalt/home/iftenney/exp/bert_mix_20190129/bert-large-uncased-lex-edges-coref-ontonotes-conll/vocab\n",
      "02/09/2019 02:25:52 - INFO - allennlp.data.vocabulary -   Loading token dictionary from /nfs/jsalt/home/iftenney/exp/bert_mix_20190129/bert-large-uncased-lex-edges-coref-ontonotes-conll/vocab.\n",
      "02/09/2019 02:25:52 - INFO - root -   Loading predictions from /nfs/jsalt/home/iftenney/exp/bert_mix_20190129/bert-large-uncased-lex-edges-coref-ontonotes-conll/run/edges-coref-ontonotes-conll_val.json\n",
      "02/09/2019 02:25:53 - INFO - root -   Loading vocabulary from /nfs/jsalt/home/iftenney/exp/bert_mix_20190129/bert-large-uncased-mix-edges-coref-ontonotes-conll/vocab\n",
      "02/09/2019 02:25:53 - INFO - allennlp.data.vocabulary -   Loading token dictionary from /nfs/jsalt/home/iftenney/exp/bert_mix_20190129/bert-large-uncased-mix-edges-coref-ontonotes-conll/vocab.\n",
      "02/09/2019 02:25:53 - INFO - root -   Loading predictions from /nfs/jsalt/home/iftenney/exp/bert_mix_20190129/bert-large-uncased-mix-edges-coref-ontonotes-conll/run/edges-coref-ontonotes-conll_val.json\n"
     ]
    }
   ],
   "source": [
    "p0 = load_task_preds(task_name, exp_types[0])\n",
    "p1 = load_task_preds(task_name, exp_types[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The cell below will score the predictions; this might take a couple minutes to run on larger datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "02/09/2019 02:25:54 - INFO - root -   Generating long-form target DataFrame. May be slow... \n",
      "02/09/2019 02:25:54 - INFO - root -   span2 detected; adding span_distance to long-form DataFrame.\n",
      "02/09/2019 02:25:54 - INFO - root -   Done!\n",
      "100%|██████████| 5044/5044 [00:07<00:00, 689.80it/s]\n",
      "02/09/2019 02:26:01 - INFO - root -   Generating long-form target DataFrame. May be slow... \n",
      "02/09/2019 02:26:01 - INFO - root -   span2 detected; adding span_distance to long-form DataFrame.\n",
      "02/09/2019 02:26:01 - INFO - root -   Done!\n",
      "100%|██████████| 5044/5044 [00:07<00:00, 695.68it/s]\n"
     ]
    }
   ],
   "source": [
    "def score_by_example(df):\n",
    "    # Score targets, but grouped on example index\n",
    "    gb = df.groupby(by='ex_idx')\n",
    "    records = []\n",
    "    for key, idxs in tqdm(gb.groups.items()):\n",
    "        sub_df = df.loc[idxs]\n",
    "        record = analysis.Predictions.score_long_df(sub_df)\n",
    "        record['ex_idx'] = key\n",
    "        records.append(record)\n",
    "    score_df = pd.DataFrame.from_records(records)\n",
    "\n",
    "    score_df['precision'] = analysis.get_precision(score_df).fillna(value=1.0)\n",
    "    score_df['recall'] = analysis.get_recall(score_df).fillna(value=1.0)\n",
    "    score_df['f1'] = analysis.get_f1(score_df).fillna(value=0.0)\n",
    "    \n",
    "    return score_df\n",
    "\n",
    "s0 = score_by_example(p0.target_df_long)\n",
    "s1 = score_by_example(p1.target_df_long)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Find examples with a large gain from base -> expt\n",
    "\n",
    "We'll group targets by the input example (`ex_idx`) and look at the scores across the whole example. Each of the cells below should print a filtered set of example indices that might be worth looking at.\n",
    "\n",
    "`mdf` merges the per-example scores from base and expt runs (i.e. `exp_types[0]` and `exp_types[1]`) so we can look for sentences where, for example, the lexical model does poorly but the full-context model gets most of the targets right."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "mdf = pd.merge(s0, s1, how='inner', on='ex_idx', suffixes=(\"_base\", \"_expt\"))\n",
    "\n",
    "mdf['f1_delta'] = mdf[\"f1_expt\"] - mdf[\"f1_base\"]\n",
    "mdf['abs_f1_delta'] = mdf['f1_delta'].map(np.abs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ex_idx</th>\n",
       "      <th>fn_count_base</th>\n",
       "      <th>fp_count_base</th>\n",
       "      <th>tn_count_base</th>\n",
       "      <th>tp_count_base</th>\n",
       "      <th>precision_base</th>\n",
       "      <th>recall_base</th>\n",
       "      <th>f1_base</th>\n",
       "      <th>fn_count_expt</th>\n",
       "      <th>fp_count_expt</th>\n",
       "      <th>tn_count_expt</th>\n",
       "      <th>tp_count_expt</th>\n",
       "      <th>precision_expt</th>\n",
       "      <th>recall_expt</th>\n",
       "      <th>f1_expt</th>\n",
       "      <th>f1_delta</th>\n",
       "      <th>abs_f1_delta</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1485</th>\n",
       "      <td>1485</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1.00</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2496</th>\n",
       "      <td>2496</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1.00</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4167</th>\n",
       "      <td>4167</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1.00</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2298</th>\n",
       "      <td>2298</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1.00</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>658</th>\n",
       "      <td>658</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1.00</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1302</th>\n",
       "      <td>1302</td>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "      <td>1.00</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3862</th>\n",
       "      <td>3862</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>0.75</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.857143</td>\n",
       "      <td>0.857143</td>\n",
       "      <td>0.857143</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4570</th>\n",
       "      <td>4570</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.166667</td>\n",
       "      <td>0.166667</td>\n",
       "      <td>0.166667</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "      <td>1.00</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.833333</td>\n",
       "      <td>0.833333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>854</th>\n",
       "      <td>854</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1.00</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.666667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2308</th>\n",
       "      <td>2308</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1.00</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.666667</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      ex_idx  fn_count_base  fp_count_base  tn_count_base  tp_count_base  \\\n",
       "1485    1485              3              3              0              0   \n",
       "2496    2496              3              3              0              0   \n",
       "4167    4167              3              3              0              0   \n",
       "2298    2298              3              3              0              0   \n",
       "658      658              3              3              0              0   \n",
       "1302    1302              6              6              0              0   \n",
       "3862    3862              3              3              0              0   \n",
       "4570    4570              5              5              1              1   \n",
       "854      854              2              2              1              1   \n",
       "2308    2308              2              2              1              1   \n",
       "\n",
       "      precision_base  recall_base   f1_base  fn_count_expt  fp_count_expt  \\\n",
       "1485        0.000000     0.000000  0.000000              0              0   \n",
       "2496        0.000000     0.000000  0.000000              0              0   \n",
       "4167        0.000000     0.000000  0.000000              0              0   \n",
       "2298        0.000000     0.000000  0.000000              0              0   \n",
       "658         0.000000     0.000000  0.000000              0              0   \n",
       "1302        0.000000     0.000000  0.000000              0              0   \n",
       "3862        0.000000     0.000000  0.000000              0              1   \n",
       "4570        0.166667     0.166667  0.166667              0              0   \n",
       "854         0.333333     0.333333  0.333333              0              0   \n",
       "2308        0.333333     0.333333  0.333333              0              0   \n",
       "\n",
       "      tn_count_expt  tp_count_expt  precision_expt  recall_expt   f1_expt  \\\n",
       "1485              3              3            1.00          1.0  1.000000   \n",
       "2496              3              3            1.00          1.0  1.000000   \n",
       "4167              3              3            1.00          1.0  1.000000   \n",
       "2298              3              3            1.00          1.0  1.000000   \n",
       "658               3              3            1.00          1.0  1.000000   \n",
       "1302              6              6            1.00          1.0  1.000000   \n",
       "3862              2              3            0.75          1.0  0.857143   \n",
       "4570              6              6            1.00          1.0  1.000000   \n",
       "854               3              3            1.00          1.0  1.000000   \n",
       "2308              3              3            1.00          1.0  1.000000   \n",
       "\n",
       "      f1_delta  abs_f1_delta  \n",
       "1485  1.000000      1.000000  \n",
       "2496  1.000000      1.000000  \n",
       "4167  1.000000      1.000000  \n",
       "2298  1.000000      1.000000  \n",
       "658   1.000000      1.000000  \n",
       "1302  1.000000      1.000000  \n",
       "3862  0.857143      0.857143  \n",
       "4570  0.833333      0.833333  \n",
       "854   0.666667      0.666667  \n",
       "2308  0.666667      0.666667  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find examples with more targets\n",
    "mdf[mdf['tp_count_expt'] >= 3].sort_values(by=\"f1_delta\", ascending=False).head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/share/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:5: FutureWarning: 'idx' is both an index level and a column label.\n",
      "Defaulting to column, but this will raise an ambiguity error in a future version\n",
      "  \"\"\"\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ex_idx</th>\n",
       "      <th>fn_count_base</th>\n",
       "      <th>fp_count_base</th>\n",
       "      <th>tn_count_base</th>\n",
       "      <th>tp_count_base</th>\n",
       "      <th>precision_base</th>\n",
       "      <th>recall_base</th>\n",
       "      <th>f1_base</th>\n",
       "      <th>fn_count_expt</th>\n",
       "      <th>fp_count_expt</th>\n",
       "      <th>tn_count_expt</th>\n",
       "      <th>tp_count_expt</th>\n",
       "      <th>precision_expt</th>\n",
       "      <th>recall_expt</th>\n",
       "      <th>f1_expt</th>\n",
       "      <th>f1_delta</th>\n",
       "      <th>abs_f1_delta</th>\n",
       "      <th>idx</th>\n",
       "      <th>num_tokens</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1485</th>\n",
       "      <td>1485</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1485</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>658</th>\n",
       "      <td>658</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>658</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1868</th>\n",
       "      <td>1868</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>1868</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2562</th>\n",
       "      <td>2562</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>2562</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2308</th>\n",
       "      <td>2308</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>2308</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2303</th>\n",
       "      <td>2303</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>2303</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1817</th>\n",
       "      <td>1817</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>1817</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2138</th>\n",
       "      <td>2138</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>2138</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2039</th>\n",
       "      <td>2039</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>2039</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1284</th>\n",
       "      <td>1284</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>1284</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>161</th>\n",
       "      <td>161</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>161</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1907</th>\n",
       "      <td>1907</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>1907</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2112</th>\n",
       "      <td>2112</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>2112</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1564</th>\n",
       "      <td>1564</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>1564</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2468</th>\n",
       "      <td>2468</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>2468</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1712</th>\n",
       "      <td>1712</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>1712</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3204</th>\n",
       "      <td>3204</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>3204</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1717</th>\n",
       "      <td>1717</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>1717</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2400</th>\n",
       "      <td>2400</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>2400</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2383</th>\n",
       "      <td>2383</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>2383</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1640</th>\n",
       "      <td>1640</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>1640</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1664</th>\n",
       "      <td>1664</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>1664</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2377</th>\n",
       "      <td>2377</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>2377</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1553</th>\n",
       "      <td>1553</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>1553</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2382</th>\n",
       "      <td>2382</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>2382</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294</th>\n",
       "      <td>294</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>294</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2827</th>\n",
       "      <td>2827</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>2827</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2831</th>\n",
       "      <td>2831</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>2831</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2833</th>\n",
       "      <td>2833</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>2833</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1023</th>\n",
       "      <td>1023</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>1023</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      ex_idx  fn_count_base  fp_count_base  tn_count_base  tp_count_base  \\\n",
       "1485    1485              3              3              0              0   \n",
       "658      658              3              3              0              0   \n",
       "1868    1868              4              4              2              2   \n",
       "2562    2562              2              2              1              1   \n",
       "2308    2308              2              2              1              1   \n",
       "2303    2303              4              4              2              2   \n",
       "1817    1817              2              2              1              1   \n",
       "2138    2138              2              2              1              1   \n",
       "2039    2039              2              2              1              1   \n",
       "1284    1284              2              2              1              1   \n",
       "161      161              2              2              1              1   \n",
       "1907    1907              3              3              3              3   \n",
       "2112    2112              3              3              3              3   \n",
       "1564    1564              1              1              2              2   \n",
       "2468    2468              1              1              2              2   \n",
       "1712    1712              1              1              2              2   \n",
       "3204    3204              1              1              2              2   \n",
       "1717    1717              1              1              2              2   \n",
       "2400    2400              1              1              2              2   \n",
       "2383    2383              1              1              2              2   \n",
       "1640    1640              1              1              2              2   \n",
       "1664    1664              1              1              2              2   \n",
       "2377    2377              1              1              2              2   \n",
       "1553    1553              1              1              2              2   \n",
       "2382    2382              1              1              2              2   \n",
       "294      294              1              1              2              2   \n",
       "2827    2827              1              1              2              2   \n",
       "2831    2831              1              1              2              2   \n",
       "2833    2833              1              1              2              2   \n",
       "1023    1023              1              1              2              2   \n",
       "\n",
       "      precision_base  recall_base   f1_base  fn_count_expt  fp_count_expt  \\\n",
       "1485        0.000000     0.000000  0.000000              0              0   \n",
       "658         0.000000     0.000000  0.000000              0              0   \n",
       "1868        0.333333     0.333333  0.333333              0              0   \n",
       "2562        0.333333     0.333333  0.333333              0              0   \n",
       "2308        0.333333     0.333333  0.333333              0              0   \n",
       "2303        0.333333     0.333333  0.333333              0              0   \n",
       "1817        0.333333     0.333333  0.333333              0              0   \n",
       "2138        0.333333     0.333333  0.333333              0              0   \n",
       "2039        0.333333     0.333333  0.333333              0              0   \n",
       "1284        0.333333     0.333333  0.333333              0              0   \n",
       "161         0.333333     0.333333  0.333333              0              0   \n",
       "1907        0.500000     0.500000  0.500000              0              0   \n",
       "2112        0.500000     0.500000  0.500000              0              0   \n",
       "1564        0.666667     0.666667  0.666667              0              0   \n",
       "2468        0.666667     0.666667  0.666667              0              0   \n",
       "1712        0.666667     0.666667  0.666667              0              0   \n",
       "3204        0.666667     0.666667  0.666667              0              0   \n",
       "1717        0.666667     0.666667  0.666667              0              0   \n",
       "2400        0.666667     0.666667  0.666667              0              0   \n",
       "2383        0.666667     0.666667  0.666667              0              0   \n",
       "1640        0.666667     0.666667  0.666667              0              0   \n",
       "1664        0.666667     0.666667  0.666667              0              0   \n",
       "2377        0.666667     0.666667  0.666667              0              0   \n",
       "1553        0.666667     0.666667  0.666667              0              0   \n",
       "2382        0.666667     0.666667  0.666667              0              0   \n",
       "294         0.666667     0.666667  0.666667              0              0   \n",
       "2827        0.666667     0.666667  0.666667              0              0   \n",
       "2831        0.666667     0.666667  0.666667              0              0   \n",
       "2833        0.666667     0.666667  0.666667              0              0   \n",
       "1023        0.666667     0.666667  0.666667              0              0   \n",
       "\n",
       "      tn_count_expt  tp_count_expt  precision_expt  recall_expt  f1_expt  \\\n",
       "1485              3              3             1.0          1.0      1.0   \n",
       "658               3              3             1.0          1.0      1.0   \n",
       "1868              6              6             1.0          1.0      1.0   \n",
       "2562              3              3             1.0          1.0      1.0   \n",
       "2308              3              3             1.0          1.0      1.0   \n",
       "2303              6              6             1.0          1.0      1.0   \n",
       "1817              3              3             1.0          1.0      1.0   \n",
       "2138              3              3             1.0          1.0      1.0   \n",
       "2039              3              3             1.0          1.0      1.0   \n",
       "1284              3              3             1.0          1.0      1.0   \n",
       "161               3              3             1.0          1.0      1.0   \n",
       "1907              6              6             1.0          1.0      1.0   \n",
       "2112              6              6             1.0          1.0      1.0   \n",
       "1564              3              3             1.0          1.0      1.0   \n",
       "2468              3              3             1.0          1.0      1.0   \n",
       "1712              3              3             1.0          1.0      1.0   \n",
       "3204              3              3             1.0          1.0      1.0   \n",
       "1717              3              3             1.0          1.0      1.0   \n",
       "2400              3              3             1.0          1.0      1.0   \n",
       "2383              3              3             1.0          1.0      1.0   \n",
       "1640              3              3             1.0          1.0      1.0   \n",
       "1664              3              3             1.0          1.0      1.0   \n",
       "2377              3              3             1.0          1.0      1.0   \n",
       "1553              3              3             1.0          1.0      1.0   \n",
       "2382              3              3             1.0          1.0      1.0   \n",
       "294               3              3             1.0          1.0      1.0   \n",
       "2827              3              3             1.0          1.0      1.0   \n",
       "2831              3              3             1.0          1.0      1.0   \n",
       "2833              3              3             1.0          1.0      1.0   \n",
       "1023              3              3             1.0          1.0      1.0   \n",
       "\n",
       "      f1_delta  abs_f1_delta   idx  num_tokens  \n",
       "1485  1.000000      1.000000  1485          12  \n",
       "658   1.000000      1.000000   658          14  \n",
       "1868  0.666667      0.666667  1868          14  \n",
       "2562  0.666667      0.666667  2562          14  \n",
       "2308  0.666667      0.666667  2308          11  \n",
       "2303  0.666667      0.666667  2303          14  \n",
       "1817  0.666667      0.666667  1817          10  \n",
       "2138  0.666667      0.666667  2138          13  \n",
       "2039  0.666667      0.666667  2039          13  \n",
       "1284  0.666667      0.666667  1284          13  \n",
       "161   0.666667      0.666667   161           9  \n",
       "1907  0.500000      0.500000  1907          13  \n",
       "2112  0.500000      0.500000  2112          11  \n",
       "1564  0.333333      0.333333  1564          14  \n",
       "2468  0.333333      0.333333  2468           8  \n",
       "1712  0.333333      0.333333  1712           8  \n",
       "3204  0.333333      0.333333  3204          12  \n",
       "1717  0.333333      0.333333  1717          10  \n",
       "2400  0.333333      0.333333  2400           9  \n",
       "2383  0.333333      0.333333  2383          12  \n",
       "1640  0.333333      0.333333  1640          12  \n",
       "1664  0.333333      0.333333  1664          10  \n",
       "2377  0.333333      0.333333  2377          14  \n",
       "1553  0.333333      0.333333  1553          14  \n",
       "2382  0.333333      0.333333  2382          12  \n",
       "294   0.333333      0.333333   294           7  \n",
       "2827  0.333333      0.333333  2827          13  \n",
       "2831  0.333333      0.333333  2831          14  \n",
       "2833  0.333333      0.333333  2833           9  \n",
       "1023  0.333333      0.333333  1023          10  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find examples with shorter sentences\n",
    "edf = p0.example_df.copy()\n",
    "edf['num_tokens'] = edf['text'].map(lambda s: len(s.split()))\n",
    "edf = edf[['idx', 'num_tokens']]\n",
    "fdf = mdf.merge(edf, left_on='ex_idx', right_on=\"idx\")\n",
    "\n",
    "mask = (fdf['num_tokens'] < 15) & (fdf['tp_count_expt'] >= 3)\n",
    "fdf[mask].sort_values(by=\"f1_delta\", ascending=False).head(30)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/share/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:5: FutureWarning: 'idx' is both an index level and a column label.\n",
      "Defaulting to column, but this will raise an ambiguity error in a future version\n",
      "  \"\"\"\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ex_idx</th>\n",
       "      <th>fn_count_base</th>\n",
       "      <th>fp_count_base</th>\n",
       "      <th>tn_count_base</th>\n",
       "      <th>tp_count_base</th>\n",
       "      <th>precision_base</th>\n",
       "      <th>recall_base</th>\n",
       "      <th>f1_base</th>\n",
       "      <th>fn_count_expt</th>\n",
       "      <th>fp_count_expt</th>\n",
       "      <th>tn_count_expt</th>\n",
       "      <th>tp_count_expt</th>\n",
       "      <th>precision_expt</th>\n",
       "      <th>recall_expt</th>\n",
       "      <th>f1_expt</th>\n",
       "      <th>f1_delta</th>\n",
       "      <th>abs_f1_delta</th>\n",
       "      <th>idx</th>\n",
       "      <th>num_tokens</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>24</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>24</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3476</th>\n",
       "      <td>3476</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3476</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3207</th>\n",
       "      <td>3207</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3207</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2881</th>\n",
       "      <td>2881</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2881</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2730</th>\n",
       "      <td>2730</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2730</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2692</th>\n",
       "      <td>2692</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2692</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2113</th>\n",
       "      <td>2113</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2113</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1870</th>\n",
       "      <td>1870</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1870</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2536</th>\n",
       "      <td>2536</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2536</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1734</th>\n",
       "      <td>1734</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1734</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>423</th>\n",
       "      <td>423</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>423</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1807</th>\n",
       "      <td>1807</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1807</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4491</th>\n",
       "      <td>4491</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>4491</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3096</th>\n",
       "      <td>3096</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>-0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>3096</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>79</th>\n",
       "      <td>79</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>-0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>79</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>130</th>\n",
       "      <td>130</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>-0.666667</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>130</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3066</th>\n",
       "      <td>3066</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>-0.666667</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>3066</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3866</th>\n",
       "      <td>3866</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3866</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1512</th>\n",
       "      <td>1512</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1512</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>109</th>\n",
       "      <td>109</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>109</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3247</th>\n",
       "      <td>3247</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3247</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3651</th>\n",
       "      <td>3651</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3651</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      ex_idx  fn_count_base  fp_count_base  tn_count_base  tp_count_base  \\\n",
       "24        24              1              1              0              0   \n",
       "3476    3476              1              1              0              0   \n",
       "3207    3207              1              1              0              0   \n",
       "2881    2881              1              1              0              0   \n",
       "2730    2730              1              1              0              0   \n",
       "2692    2692              1              1              0              0   \n",
       "2113    2113              1              1              0              0   \n",
       "1870    1870              2              2              1              1   \n",
       "2536    2536              1              1              0              0   \n",
       "1734    1734              1              1              0              0   \n",
       "423      423              1              1              0              0   \n",
       "1807    1807              2              2              1              1   \n",
       "4491    4491              1              1              0              0   \n",
       "3096    3096              1              1              2              2   \n",
       "79        79              1              1              2              2   \n",
       "130      130              0              0              3              3   \n",
       "3066    3066              0              0              3              3   \n",
       "3866    3866              0              0              1              1   \n",
       "1512    1512              0              0              1              1   \n",
       "109      109              0              0              1              1   \n",
       "3247    3247              0              0              1              1   \n",
       "3651    3651              0              0              1              1   \n",
       "\n",
       "      precision_base  recall_base   f1_base  fn_count_expt  fp_count_expt  \\\n",
       "24          0.000000     0.000000  0.000000              1              1   \n",
       "3476        0.000000     0.000000  0.000000              1              1   \n",
       "3207        0.000000     0.000000  0.000000              1              1   \n",
       "2881        0.000000     0.000000  0.000000              1              1   \n",
       "2730        0.000000     0.000000  0.000000              1              1   \n",
       "2692        0.000000     0.000000  0.000000              1              1   \n",
       "2113        0.000000     0.000000  0.000000              1              1   \n",
       "1870        0.333333     0.333333  0.333333              2              2   \n",
       "2536        0.000000     0.000000  0.000000              1              1   \n",
       "1734        0.000000     0.000000  0.000000              1              1   \n",
       "423         0.000000     0.000000  0.000000              1              1   \n",
       "1807        0.333333     0.333333  0.333333              2              2   \n",
       "4491        0.000000     0.000000  0.000000              1              1   \n",
       "3096        0.666667     0.666667  0.666667              2              2   \n",
       "79          0.666667     0.666667  0.666667              2              2   \n",
       "130         1.000000     1.000000  1.000000              2              2   \n",
       "3066        1.000000     1.000000  1.000000              2              2   \n",
       "3866        1.000000     1.000000  1.000000              1              1   \n",
       "1512        1.000000     1.000000  1.000000              1              1   \n",
       "109         1.000000     1.000000  1.000000              1              1   \n",
       "3247        1.000000     1.000000  1.000000              1              1   \n",
       "3651        1.000000     1.000000  1.000000              1              1   \n",
       "\n",
       "      tn_count_expt  tp_count_expt  precision_expt  recall_expt   f1_expt  \\\n",
       "24                0              0        0.000000     0.000000  0.000000   \n",
       "3476              0              0        0.000000     0.000000  0.000000   \n",
       "3207              0              0        0.000000     0.000000  0.000000   \n",
       "2881              0              0        0.000000     0.000000  0.000000   \n",
       "2730              0              0        0.000000     0.000000  0.000000   \n",
       "2692              0              0        0.000000     0.000000  0.000000   \n",
       "2113              0              0        0.000000     0.000000  0.000000   \n",
       "1870              1              1        0.333333     0.333333  0.333333   \n",
       "2536              0              0        0.000000     0.000000  0.000000   \n",
       "1734              0              0        0.000000     0.000000  0.000000   \n",
       "423               0              0        0.000000     0.000000  0.000000   \n",
       "1807              1              1        0.333333     0.333333  0.333333   \n",
       "4491              0              0        0.000000     0.000000  0.000000   \n",
       "3096              1              1        0.333333     0.333333  0.333333   \n",
       "79                1              1        0.333333     0.333333  0.333333   \n",
       "130               1              1        0.333333     0.333333  0.333333   \n",
       "3066              1              1        0.333333     0.333333  0.333333   \n",
       "3866              0              0        0.000000     0.000000  0.000000   \n",
       "1512              0              0        0.000000     0.000000  0.000000   \n",
       "109               0              0        0.000000     0.000000  0.000000   \n",
       "3247              0              0        0.000000     0.000000  0.000000   \n",
       "3651              0              0        0.000000     0.000000  0.000000   \n",
       "\n",
       "      f1_delta  abs_f1_delta   idx  num_tokens  \n",
       "24    0.000000      0.000000    24           9  \n",
       "3476  0.000000      0.000000  3476          19  \n",
       "3207  0.000000      0.000000  3207          14  \n",
       "2881  0.000000      0.000000  2881           4  \n",
       "2730  0.000000      0.000000  2730          13  \n",
       "2692  0.000000      0.000000  2692          11  \n",
       "2113  0.000000      0.000000  2113          14  \n",
       "1870  0.000000      0.000000  1870          18  \n",
       "2536  0.000000      0.000000  2536          12  \n",
       "1734  0.000000      0.000000  1734          12  \n",
       "423   0.000000      0.000000   423          14  \n",
       "1807  0.000000      0.000000  1807          18  \n",
       "4491  0.000000      0.000000  4491          19  \n",
       "3096 -0.333333      0.333333  3096          11  \n",
       "79   -0.333333      0.333333    79           7  \n",
       "130  -0.666667      0.666667   130          17  \n",
       "3066 -0.666667      0.666667  3066          19  \n",
       "3866 -1.000000      1.000000  3866          16  \n",
       "1512 -1.000000      1.000000  1512          15  \n",
       "109  -1.000000      1.000000   109          12  \n",
       "3247 -1.000000      1.000000  3247           9  \n",
       "3651 -1.000000      1.000000  3651          15  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find examples with no change\n",
    "edf = p0.example_df.copy()\n",
    "edf['num_tokens'] = edf['text'].map(lambda s: len(s.split()))\n",
    "edf = edf[['idx', 'num_tokens']]\n",
    "fdf = mdf.merge(edf, left_on='ex_idx', right_on=\"idx\")\n",
    "\n",
    "mask = (fdf['num_tokens'] < 20) \n",
    "mask &= (fdf['tp_count_expt'] <= 2)\n",
    "# mask &= fdf[\"tp_count_base\"] <= 1\n",
    "mask &= fdf['f1_expt'] <= 0.5\n",
    "fdf[mask].sort_values(by=\"abs_f1_delta\", ascending=True).head(30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plot the diffs on these examples\n",
    "\n",
    "The `ex_idx` column printed in each of the above cells is the index of that example in the lists of raw predictions. Enter it in the cell below to show the predictions (and ground truth) on base and expt for that example.\n",
    "\n",
    "By default, it will print all predicted classes with `p(class) > 0.5`, and give their predicted probabilities in parentheses."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Text (14): the soldier was a religious man , one of his close help ##ers .\n",
      "\n",
      "  span1: [ 0, 2)\t\"the soldier\"\n",
      "  span2: [ 9,10)\t\"his\"\n",
      "  label: (1)\t\t 0\n",
      "  pred:  \t\t 1 (0.92)\n",
      "\n",
      "Text (14): the soldier was a religious man , one of his close help ##ers .\n",
      "\n",
      "  span1: [ 0, 2)\t\"the soldier\"\n",
      "  span2: [ 9,10)\t\"his\"\n",
      "  label: (1)\t\t 0\n",
      "  pred:  \t\t 1 (0.88)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# ex_idx = 1868 # \"And their important leaders ...\"\n",
    "# ex_idx = 1767 # \"Jesus looked at the man ...\"\n",
    "# ex_idx = 2305 # \"He would not stop to help him either .\"\n",
    "# ex_idx = 3476 # \"It wasn't until Lee Teng - Hui ...\"\n",
    "# ex_idx = 4031 # only one local ringer\n",
    "# ex_idx = 1734 # book of life\n",
    "# ex_idx = 3066 # martha stewart\n",
    "# ex_idx = 3866 # among its new customers\n",
    "# ex_idx = 3651 # Tien stressed that the president ...\n",
    "ex_idx = 2113\n",
    "print(analysis.EdgeProbingExample(r0[ex_idx], label_vocab=p0.all_labels))\n",
    "print(analysis.EdgeProbingExample(r1[ex_idx], label_vocab=p1.all_labels))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
